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>' - ^ : , / The Effect bf Training 

Whe Effect of Tcainl^g/Supefyision -of Nonproi'e/sional 
Interventions with. Delinquents 

The last two decades have seen a '*ray id .increase in the nlimbers of programs 



1 



using 'paraprofession^lsr in fact, the use of nonprofessionals and volunteers ' 

' ' ... ' ' . 

haigp been a'mai^or characteristic of the development of the community psychology 
Tqiovemejit /([Rappaport, 1911; Hejler & Monahan,: 1977; Nie^zel, Winett, MacDonald, 
V & ,t),av/3fdsoh,'-1^77) . Early reviews of the research in this area were positive, 

spjA^re^d^ by sjuch controversial studies as Poser (1966)- and Rappaport^ Chinsky and 
Cowen 1(^974) , but noted the preponderance of simple outcome studies and . the need 
for systei^tlc research during the 1970's. The majority of the .more recent non- 
. prof ess'ton^ studies' have " continued to look at basic outcome questions ra,ther than 

at separation of program components, e .g. , training^ supervis^ion, intervention 

- ' .-^^ ^ . f ' ^ . , ' - ' ' . ■ ' 

strategies — and' hive done so without using objective measures from the p'er^^pec tive 
of tariget .groups <Rappaport, 1977; Ziippfer, 1974). In fact, training, supervision, 
'and"^ interverkion variable5;'h6v^' of ten b^en so .confounded that the differential^, 

influence of various nonprbf eiss^bnall jJrogirara components could not be determined. 

■ ■. J- ■ ' ■ . ^ ■ , r 

Since the early erithy^^ias^ about tl^ use j^f ' itoriprof essionals in various social 
problem areas there ha^ be^n. a Jjaupity of r^eseart either basic out^tne- 

! ' ■v"^' ■ ■ 

or differential efficacy dimensions;- Enthusiasm, for nonprofessional programs 

• . . ■ ' ^ ^ ' ' ■ ' ' ' . " ^ / 

^ continues to run high, and the numbeV of programs continues ^to increase. However, 

conclusions about paraprof essional effectiveness have often been. based on programs 

in which numerous variables , such as the content, format* and amount of training, 

supervision, and intervention were highly confounded. As well, the evidence that - 

, hias been obtained is often indirect, subjective in nature, and not drawn from 

rigorous research designs. In addition^ a significant lack in research to date 

has been the inclusion of variables from the perspective of tl^ i:.ecipients of^ 

nonprofessional inte^ention. ^ - . ^ 
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' - : \ 
' ' * •/ ^ ' ^• . . ■ 

The typical program reported in tfije literature has' been monolithic in its 
/ • . - . • 

, interventtbW ^o;cus and rathe^r narrow in. the dependent variables examined. For 

> / ^ ' c 

example, the tdaj or "schools of 'though't'V regai^ding change tactics have all appeared 

' ^\ ' ' C ,/ •' ^' 

in* the nonprofessional literature. These have included the use of essentialLy 

"untraiiied'; college students, use of college stijiden^ ^trained in^ Rog^rian based 

skills, nonprofessional behavior analysts', and more recently hordes- *bf ' child 

advocatefe. The relktlve effectiveness of such procedures and the salient processes 

. . r * ^ ' ) " ' ' . 0 

had be^>erntire*ly igj)pre'd. • - w /' . / I , . / 

r---'-^he aim of the current study was to presen^\ preliminary findings which will 

examine and contrast the'^effects of four different strategies of training/super- 

■ . " ' ' . ^ / / . Vf 

vision/intervention usd'd within .the_CQfij:ext of a diversion Program for iuvenile 

rdelinquents. Multiple measures were used to examine issues related to the results 

^ ' I ■ \ . ' y ^ ' ^ - ^ ^ ■ ' 

of these training mterveiv^^lon and-^ypei^vision strategies. a critical component- 

of ^e study was the development ^pf pr*^2S^'^T]ieasures vhich ijicluded'^ data 

* -"V ' ■ ■ " V \ - ' 

from the recipients of the nonprofessional, interventioi>^. The iriajor isstre ^examined 

in this study was a description of the, performance* of student volunteers .trained / * 

in, four different intervention. i^dali/tiesl The central fbcus was ttie actyjaj per-> 

formance of the students in the naturalistic setting ratiei^ than in a laboratory^ 

analogue. The f^ur sp^c^fic models/ to be compared' involved high iptejisity 

training in the methods*^of: behavio/al contracting' and chilci . advocacy , high inten- 



sity training in the methods of relationship skills., low . intensity training f ' " 
ccomplished in large groups^, andj low intensity training: accomplished^in sijiail 



groups. Obviously at issue, were tJie format .(intensity) and content^ oi . training. ( 
A 'sectsnd -tnajor focus of thi$ study was the development of inlterview biased process 
measures fhr examination of^-^t^e actual interventions pi-pVided by the student 
v<^lunteers., / ^ „ - 
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METHOD 



Context of ^the Research 




This^ study tcJok place within the context df a larger research project\ funded 
. by the National Institute of Mental Health. The overall project has bden it 
, formal operation for approximately two.^ years and is aimed ^t replicating , and 
epcperimentally examining the effectiveness of a specific diversion pto^ram for, 
•delinquent youth (Davidson & Rapp, 1976; Davidson & Rappaport, 1978; Davidson, 
^5eidman, Rappaport, Rapp, Berck, Rhodes, & Herring, 1977; Seidman, Rap^port.& 
Davidson, 1976). ^ The overall ope^rat ion of the project involved college students 
working on a one-to-one basis with youth diverted from a local juvenile court . The 
overall research project' examined silch components of the divet^ion model as the 
degree of involvement of the youth in the ^juvenile justice system at the time (of 
rel 



ferral, ^l^^acteristic^f vthe college students, selection of nonprofessionals, 
and the scope of the intervenjtdom approach, 

' T^e entire program operated under the e^catioital pyramid model prescribed by 
Seidman a^d Rappaport (1974). The ■ pri^cip^l investigator supervised, trained, and 
consulted with nine graduate atiA ^two |Hdergraduati^<,studeilts who 'have the responsi- 
bility fpr training and supervising small groups^. of undergraduates and for coordina 
ting project; research, The nonprofessional trd±hing*,aiid supervision segment of 
the project was a formal three term undergraduate psychology course it: Michigan* 
State University, ^ ' 

In addition to the project's relationship with the University^nd the Depart^ 
menr of Psycholggy, tHere were close ties with the locaj juvenile justice system. 

The local juvenile court currently handles approximately five to six hundred 
,, ' V/ ^ * ' . 1 ■ 

youthful offenders annually. It is from this project group tha( referrals were 

made, ^ f * ' ■ ' ' 

/ • ' - . ■ . 

Seventy-seven delinquent] youth w^ere referred ;to-the project from October,' 1977 
though January, 1978. Four of the youth decided that -they did not want 
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participate. Those youth who decided to voluntarlTy participate ifi the jJroject 

demographic characteriost ics : 62 mal^s and ^1 females j 51 whites 
and 22 nonwhites; the average youth was in The ninth grade with a range of 5th ^ 
to 12th; 65% of the youth came from broken homes; and the average age. was 14.3. 
Criminal activities ranged across the full spectrum, but tended to be nonserious 
and serious misdemeanors and nonserious felonies. There were no status offenders 
among the group -re terred to the project. There were no significant differences ' 
among the youth assigned to the four experimental conditions or the }bontrol group 



on any of the demographic variables. 



Subjects ' 

Delinq'uent Y outh J ^ 

Stratified by court referee, race, sex, formalAnf ormal handling, in order 6f 
referral, the 73 youth who volunteered to participate were randomly assigned to 
one of the four experimental groups or to a control condition .(control subjects 
were, not at issue in the portion of the research repprted ^n this paper). At the. 
outset, 14 youth were assigned to.the Hi^ Intensity/Behavioral Contracting/ 
Advocacy Condition, 13 \o the High Intensity/Relatior/ship Condition; 15 to the Low 
Intensity/Small Group ^ondition, 16 to J:he Low Intensity/Large Group, and'^16W 
' the Control Condition. There were^no significant pre differences on any of ^ the 
'^vaxiabXes used in'' t:he overall study between .the five gi^o.ups. For the particular ' 
aspect of the overall research reported in this^aper, the 57 experimen£al. youth 
are of concern, v Of ^he 57 experimental youth( 5% were ^^vailable for complete 
interview dat^ collection during, the study . The attritiOTr of three subjects \ ^ 

occurred at a rate of one each in threi of ttiid experimental .conditions and ij^i 
result of inaccessibility t;,o the interviewer. f 



Nonprofessi onal Volunteers ' . ' ' ^^'^^ 



/""^ The nonprofessional volunteers used itv t^is' study were sele^^d from a iafee" 
group Of undergraduate, college sophomores, ju^itjrs, and seniors. In the springs of 
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197^ a letter was sent to 3,500 undergraduate social science majors at Michigan 
- ^tate. University.' This'^etter informed them.,^ the j>osiiibility for participation 
±n the project, provided a bri^f description of the program, and invited them to 
call the project office if they were intere^^ed. -Flrom this number, approximately 
400. called 'lihe project office and ^xpre^sed\lnterest in participation:. Of the 
: , group that caMed, 300 attended ap iniUal '^i^d^ng. Xl^i^^, initial meeting was for 
the. purpose of descVib;Lng in iore det^n the'^i^i^iAons for project involvement 



, and the requirement for participation term course sequence. At the 

end of this sesfeion, approximately^^g stlidents agreed to proceed in the selection 



, process. Fc^lloWing the initial^me^ing, potential volunteers were required to 

' ■■ r ' • ' ^ • . • 

>? rfetui^^or an additional assessment sessipji'V This second meeting consisted of* 
' t^^^!^^ '^^^ noHprofrfssiopa.l volunteers jfon^^lete approximately three hours of paper 
y^^^?^* pencil measur/s being used '^as part l)f the selection and student outcome aspects 

of the research. 130 nonprofessional A^g|unteers completed this entire process 
I arlid were eligibly for project participation. Of the 130, 57 were randomly 
sel«^ted for pa^^^patidn in the four experimental conditions. The 57 student A " 
'^.^^^^^^^ were ^'assigned to the four experimental conditions randomly with strati-> 
^iq^tion for s<&x. Comparison of pre measures on over 70 variables failjed to reveal 
' significant differenced in any of the four groups, above what would have been 
expected i)y chance. ' ^ - 

rrainerV§\ipervisors - ' ' , . 

There were ten trainer/supervisors involved in the training and supervision 
of the students -ill the four experimental conditions. Seven of the trainer/super- 
visory were* advanced graduate students in psychology with major interests in the 
area of copmunity/cl!iriical or ecological psychology. The eighth trainer/supervisor 

4was a senior majoring in psychology. This individual has been a nonprofessional 

# 

>^ volunteer in-'preyious phases of the project and had also served in other research 
assi St an;^'*''roles. These first eight trainer/supervisors were responsible for 
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conducting the training and supervision groups in the High Intensity Behavioral 

y ' - , 

Contracting/Cl^ild Advocacy Condition and Re^lat ionship Skills Condition. 

The ninth and tenth trainer/supervisors were responsible for conducting the 

two Low Intensity ^Conditions. One of these two was a senior majoring in social 

work who was a staff member for the Student Volunteer Bureau at Michigan 'state 

'■ / ^ . . • , * 

University-. She had been a volunteer and volunteer student coordinator, assuming 

management responsibilities, at a shelter resideiTce for delinquent apd abused 

children. In addition, she had done field placement work in the'child care area. 

. yhe second Low Intensity trainer/supervisor was an advanced graduate student in 

social work with major interests in program planning and 'administration. She had 

been a student volunteer in' a number of programs, a staff member in a youth service 

bureau, ani the director of volunteer in court progi^am. During the year in which 

sh,e served as a trainer/supervisor 'for the project, she also served as a staff 

lunteer, directing several programs for the Michigan State University' Volunteer 



vo 

41 

Bureau. V 



s 



All trainers and supervisors in the high intensity classes (both relation^ip 
kills and contracting/advocacy) were- assigned to condition by choice. In addition, 
the particular skills and background of the trainers and supervisors were taken 
into account in assigning them to experimental conditions. The background and 
experience of the trainer/supervisors for the two low intensity conditions were 
particularly relevant since they had the type of program experience that this study 
was attempting to duplicate in the low intensity conditions. 

Research Design 

The aspect of the research r^eported here included a single experimental design. 
In order to examine questions concerning the impact of training/supervision/inter- 
vention condition throughout the project's experience, the research design was 
a four X three design with repeated measures. The two dimensions were training 
cc^ndition (four) x time (six weeks following referral, twelve weeks following 

er|c ' £; 
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referral, and termination — 18 weeks following Referral), Process interview 
measures, to be described in a later section, were administered to deferred youth, 
fehei-r parents, nominated peers, and their assigned volunteer at each of the* ^ 
three time periods. In addition to this basic four x three design, an additional 

i 

dimension, success/failure, -resulted in the four x three x two analyses to be 
ft 

presented in this paper. 

Procedures 

Referral of Delinquent Youth 

Youth were referred to the diversion project by the intake staff of the local 
juvenile court. The agreement that the project had with the staff was that they 
would refer youth to the project for whom they would otherwise consider for "formal" 
processing, fn Other words, the intake staff agreed that they would not refer 
youth to the diversion project who would otherwise be warned and released. At'the 
ot^er end of the spectrum, the court staff did not refer serious offenders (major 
crimes against person) to the project. After the decision was made to divert 
the youth to the Adolescent; Project, the court referee briefly explained the pro- 
gram to the youth and his/her family. If the youth and the family were interested 
In considering the diversion project as an option, a referral interview was set 
up for them with the project staff. During this initial interview, the program 
was explained in detail, the methods of assessment involved outlined, the random 
^ assignment procedure described, etc. After completing '^this introductory material, 
the youth and the parents were asked if they were willing to participate. Assum- 
\ng that they voluntarily agreed to participate in all aspects of the project, 
they committed themselves both verbally and in writing. Following the completion,.^ 
of the voluntary participation procedures, they were assigned to one of the four 
experimental conditions or th^ control condition according to procedures described 
m the Subjects section. 

ERIC ,j 
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Tralnlng/Superylslon/Interyehtion Conditions • / 

1 Students assigned to all experimental groups received four academic credits 

per term for 'the, 1977-78 academic year. They were required to attend all super- 
vision sessjpns scheduled for thefr particular training/supeTvision condition-, to' 
hand in weekly progress "reports, to keep an event logbook— a running account of - 
contacts between the youth and with others on behalf of the youth— to participate 
* in assessment procedures and the process interviews described below throu|^hout 
the three terms, and to meet with their youth six to eight hours per week for 
eighteen school weeks. "Grading for the class aspect of the project was based on 
case responsibility, class attendance, case presentation, class discussion, and 
following ethical standards. ' , ' " • * 

High Intensity training/supervision/interveny^on . Each of ,the fpur High 
Intensity classes (two High Intensity , Contracting/Advocacy , two High , Intensity 
> Relationship Skills) were composed of seven students and two graduate student 
trainer/supervisors. Each supervision sessiori met for two hours per week. The 
first. eight weeks of the Fall term of 1977 were designated as a formal training 
segment j,j^^^tudents working through programmed manuals, outside readings, and 
in class demonstrations and role plays (detailed descriptions of these procedures 

and manuals available from the authors upon request). Mastery of the subject 

/ 

matter was. demonstrated on oral ahd writ ten(weekly questions. 

After the. training component was completed, the classes were used as. general 
case supervision sessions. " Students were assigned cases as early as the fourth 
week of training an<^ others were not assigned to cases until six weeks following the 
completion of training. The structure, general format, and class size of the two 
High Intensity conditions was -.similar. However, the content of training/super- 
vision/intervention and the resulting impact on supervision philosophy and inter- 
vention execution was very difficult.' The two particular subsets of high intensity 
training/supervision/intervention are described below. 
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^) High Intensity conditions — contracting/advocacy . Students in this 

condition were trained In the^use of skills generally described as child advocacy 

(Davidson & Rapp, 1976) and behavioral contracting techniques (Stuart, 1971). 

The training sessions (eight weeks) for this condition included the following 

eight units: (1) an overview 6f the course, the juvenile justice system, and the 

theoretical rationale behind behavioral conceptions of human behavior; (2) the 

theoretical rationale underlying the child advocacy conception of human behavior, 

rationale for using a combined" advocacy and contracting approach; (3) a brief 

description of inAt ial- meetings between students and youth; a description of 

indepth behavioral and advocacy assessment; a description of modes for collecting 

assessment information on the youth's interpersonal network; methods for collecting 

■ 

informatiou relevant to the unmet needs of the youth and determination of avail- 
able community re^sources; (4) outside homework assignments and in class role plays 
^ descriptive of behavioral assessment methods, demonstration of methods for 
advocacy oriented assessment; (5) methods for selection .awtf initiation of inter- 
vention strategies, strategies for negotiating contracts, demonstration of the 
mediator role, selection of an advocacy strategy and target; (6) practice assign- 
ments, role play, and demonstrations of contracting ar>d advocacy intervention 
approaches; (7) strategies for monitoring the effects of ^^TTtracting and advocacy 
interventions; (8) preparation for termination training, description of^ practice 
sessions to be used with referred youth and significant others. 

Once the ei^ht weeks of training were complete, the two hour wcLkly class 

/I 

sessions were spent in case supervision. These case supervision sessions consisted 
of sequential presentation of each nonprofessional volunteer's case progress. 
Supervisors and classmates assisted in- a general problem solving f onnat group 
discussion including clarification of goals, review of technqiues, and specifica- 
tion of future plans. The interventions to be implemented by students in this 
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condition were expected to follow the steps outlined and modeled in the training 
and superyision segment. In short, the intervention was to proceed through the , 
phases of assessment, contracting and advocacy intervenjtlon, monitoring, anfl • 
termination. The High Intensity Clii Id Advocacy/Behavioral Contracting Condition 
is a direct replicate of intervention procedures which have demonstrated to be 
successful in previous research by the second author (Davidson et al., 1977). 

^) Hii2]i^Jjitensrtx--Rj'l*^^^^^ ^^^^ second high intensity 

training/supervision/intervention condition used a relationship skill approach. 
Design of this condition was parallel in format to condition "a" described above, 
except that the training and intervention approach taken was that of relationship 
skills (Carkhuff, 1969; Egan, 1975; Goodman, 1971). The training segment for this 
condition was parallel to that for condition '*a'' and* inc luded the following ^ight 
units: (1) an overview of the course, a brief history of the juvenile justice . 
• system, ^ and a rationale for diversion; (2) theoretical rationale for the essential 
ingredients of the therapeutic relationship including empathy, nonpossessive warmth^ 
and genuineness; (3) a bpief description of initial- meet Ings between the student 
and youth, empjtasis on skills for enhancing comfortableness, trust, etc., and the 
role of crises in developing and maintaining relationships; (4). the role of self- 
understanding and the importance of knowing and accepting feelings in intervention, 
-I »- ' 

the rationa^ for empathy skills and practice sessions and role plays in feeling 
identification and feeling oriented responses; (5) practice assignments and role 
play sessions on the skills of clarification and paraphrasing, and a description 
of commonly occurring errors; (6) the role of openness in relationships, the skills 
of behavioral description, feedback and confrontation of discrepancies; (7) the 
importance of the facilitation of autonomy, the problem solving process, the 
definition of problems and goals, and methods for the discussion of alternatives; 
(8) a summary of the model" including an overview of the total probleip solving 
process and the components of termination. 

ERIC , o 
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The case supervision structure- parallels that of condition ''a". Namely, the 
two hour weekly sessions for the remainder of the three terms were used for the 
supervision of cases. The focus throughout was on helping the students implement 
an intervention based on relationship skills during the eighteen week intervention 
period. 

i. 

Low_jn^ten sity trainifig/supervisi o n/ intervention . There were three low 
intensity classes. The two with seven students comprrised the ^ow Intensi^ Small 
Group Condition. The third, with 15 students, comprised the Low Intensity Large 
Group Condition. In many ways, these two conditions modeled the type of training/ 
supervision/intervention often presented in volunteer. in court type programs. 
"Court volunteer training is rarely elaborate, intensive, or formal. The average 
seems to be about five to ten hours, spread out over two or three consecutive 
evenings in one week or once a week over several weeks (Scheier, 1971, page 74)." 
From another point of view, these two conditions represent attention placebo 
control for the first two conditions. 

Low In_t ensity — Small Group "Condition . The training phase of this condition 
included three initial orientation meetings, held weekly during the first three 
weeks of the Fall term, 1977. The orientation meetings were primarily didactic 
in nciture. The first meeting provided a general introduction to the course, the 
require :.:ert-s for in class and field work experience, an overview of the project, 
and a rationale for diversion. The second meeting involved a lecture by the 
trai:.- : . ^^^r - i visors on prominent theories of juvenile delinquency in the juvenile 
justice system and their practical implications. The theories that were covered 
included differential association, intrapsychic explanations, interactionist 
positions, environmentalist positions. This was followed by a detailed description 
of the juvenile court system in the local community. In the final portion of 
the second meeting, the mechanism for assigning youth to a student were presented 
and a brief discussion for initial contacts took place. In the third meeting. 
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several specific topics were discussed. First, there was a general discussion 
of the kinds of activities that students could engage in vith their assigned 
youth. This included getting to know the^ youth and their problem areas, and 
-determining how to make changes and solve identified problems. The emphasis 
throughout was on \he students using their ''naturally occurring" skills -in order 
to facilitate change. In addition, the availability of information on services 
in the local community was described. The necessity of being' supportive and 
''hanging in there" with the youth, even when things got difficult, was emphasized. 
In addition, the rationale for the importance ©f the eighteen week time frame and 
preparation for termination were stressed. 

Supervision sessions in the Low Intensity Small Group Condition were tvo hours 
in length and held on a monthly basis following the initial three orientation 
meetings. They were run by the two trainer/supervisors. The overall philosophy 
in supervision was that the interest, commitment, and high level of motivation 
and enthusiasm of the voluiiteers could have a positive impact on youth without the 
need for more specific skill oriented training. Not providing a specific inter- 
vention plan allowed the opportunity for the nonprofessional volunteers to try 
out what they felt was useful, to be maximally free and flexible in whatever they 
chose to do, thus using the natural skills which they possessed when they came 
to the project. Supervision sessions were a time for idea exchanging, holding 
group problem solving sessions, and for dealing with routine administrative matters. 
Th.: supervision style adhered to in these sessions was nondirective in nature. 
Questions raised were explored by the class as a whole. The supervisor's task 
was to summarize and integrate the discussion and reflect the comments and ques- 
tions of the students. The students were encouraged to try out various alterna- 
tives in the actual intervention with the youth. 

^) Low Intensity — Large Group . The Low Intensity Large Group Condition was 
an attempt to examine the effects of carrying on nondirective training in a large 
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group. This was an attempt to reflect the size of group often used in typical 

community volunteer programs/ The low intensity/large group consisted of 15 

• ^ '6 /. * 

students meeting together In both the training and supervision segments. The 

X ; ^ . 

procedures followed in t,^ low intensity large group condition mirrored those 

followed in condition *c " above . 

Measures 

Delinquent youth . While a large number of outcome variables were gathered, 
from archival sources in the larger project of which this study is a part, two 
particular measures are relevant. As described in the section on design, the 
perf ormancer of student volunteers was examined in relationship to condition, time, 
and success/failure. In. order to determine a success/failure categorization, police 
and court records were used. The frequency of police contacts and juvenile court 
petitions were recorded for all youth for one year prior to the project and at 
varying followup intervals. For purposes of this substudy the police and court 
records collected for the time period during which the youth were involved in the 
project are of interest. Namely, youth were categorized as success cases if they 
did not have either a police contact or court petition during the time they were 

involved in the project. Failure cases were defined as any cases in which there 

3 

was a police arrest or juvenile court petition. 

Process interviews . Process interviews were conducted for the overall 
research project at ^hree times during the course of a youth's and student's 
involvement. This provided a detailed participant oriented monitoring of the 
events that occurred as a result of the training/supervision/intervention conditions. 
As mentioned earlier, ^the three time periods were six weeks, twelve weeks, and 
eighteen weeks following the matching of a student and a referred youth. These 
interviews were held with the target youth, one of his/her parents, a nominated 
peer (specified by the youth as a close friend), and the assigned volunteer. 
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The development of process interview scales was aimed at increasing our " 
empirical understanding of the operation of the program and its relationship to 

/ 

critical^ life events from the perspective of program participants. As such, .a 
/equent/al scale development procedure was undertaken which involved the generation 
k behiviorally specific items from the perspective ^f program participants (youth, 
-^parents, nominated- peers , and student volii^teers) , roUo'.ing the construction of 
such an item set, the resulting data were submitted to a sequential rational/ 
empirical scale construction procedure. The result was a series'of scales reflec- 
tive of the major characteristics of the intervention. 

In the spring of 1977, fifteen interviewers were recruited to carry out the 
process interviews during the phase of the research reported here. These inter- 
viewers were given six weeks of training in interview procedures. This included 
familiarizing them with t^jj^^sic purpose of process interviews as a strategy of 
gaining participant relevant yet behaviorally specific data on the^eperat ion of 
the project. Next they were introduced to the item set (220 ^tT^s) which had been 
developed in earlier phases of the research. They then were instructed in inter- 
viewin^^actibs to be used. This consisted of verbal instruction by two graduate 
assistants in charge of process interview research components, demonstration of 
interview strategies through tapes of previous interviews, role plays and training 
sessions, and finally practice interviews with their roommates and youth recruited 
from the local community. All practice interviews were audio taped and formed 
the basis of feedback from interview supervisors. 

Following the training, the interviewers were assigned to cases to 
conduct the actual interviews. Interviewers were responsible for interviewing the 
youth, parent, peer, and voli^teer within each case they were assigned. As indi- 
cated earlier, the interviews took place at six week intervals during a youth's 
eighteen week involvement with the project. Four complete sets of interviews 
were completed with two interviewers present in order to assess inter-interviewer 
agreement. Over all sources of information, at 
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all, time periods, and on all items, the interviewer pair showed- a 70% level of 



li t] 

4 

exact agreement demonstrating the interviewer reliability of the interview data. 

* "The intervention items were submitted to a rational empirical strategy of 
scale construction (Jackson, 1971). This involved initial Rational grouping^ •Taf 
t^e item scale-^^n^ the intervention area. The internal consistency of the^e 
rational groupings was then determined according to two criteria. First, in order 
for an item to remain on its scale, it had to show a significant correlatiop with . 
, the sum of the other items on the scale. Second, the item had to demonstrate a 
significantly higher correlation with its own scale than with other scales in the 
intervention domain. If it showed a greater degree of convergence with another 



scale it ,was moved if it mad^ rational sense. Otherwise, t^e item was discarded^'i' 



from futher analyses. This rather lengthly process of scale construction pro- 
duced scales which were maximally reliable and maximally orth9gonal. As a result 
of this process the scales described in Tab^e 1 were developed. Table 1 presents 
the conceptual content of the items on each scale, the number of items on the 
scale, and the alpha coefficient (a detailed presentation of the scale construction 
process is available from the authors upon request). The final step in the scale 
construction process was to examine the convergent and discriminant validity of 
the above described scales. The multitrait multimethod matrix prescribed by 
Campbell and Fiske (1§59) was used to examine the convergence of scales across 
data source (i.e., youth, parents, peers, and volunteers). In short, this process 
resulted , In scales which showed excellent convergent discriminant validity 
properties. As' a result, final scale scores were summed across dat^ source 
resulting in a single score on each of the fourteen scales for each subject at 

each Lime period. Id§SIE.^5^1?„I_^^2^£ ^?£^ 

RESULTS 

Given the large number of dependent measures described in the >lethods section 
and the preliminary nature of the study being reported here, only a sample qf the 

\ 



ERIC . 1' . i 



' ' ' / 



^ I , \' TSe Eifects of lic.Mjang ^ 

16 



\ results wij-l be re^orted/^in detail.^ Results from the remainder of the proces^ 

* interview scales will l^e described verbally. ,A detailed analysis of variance 

an-d figures represent^tivg., of mean levels will not be^incJ.uded due to ^spac^' 

limitatii^ns. Overall strategy for analysis of process interview scales was a ^ \ 
^ ^. / / ^ . ■ . ' ' . , ^ • \ 

f our, X 4hre^,x two Analysis of vkriajice with repeated measures. -Each of ^he process 

interview scales' was analyzed by'- condition, success vs. failure, and tiije ^'using*^ 

complex analysis jpf variance wi*th repeated measures. In.-addit ion , plapned ^ 

comparison Scheffe tests were used to examine group 0^axi differences of ^part^cu^ar 

^ interest. Again for purposes of br^^vity and clarity, exact F ^atistics tor 

\ Scheffe multiple comparisons will not be provided, all Scheffe comparisons 

described can be assumed to reach the .05 leyel of significance, ^ The results 

will be organized in two subsections. The first will present the results on the 

three condition-specific scales. Tltfese will include the* Adherence to Be^havlora^ f 

Contracting Model Scale, the Adherence to Child Advocacy Model Scale, and 

Adherence to Relationship Skills Model Scale. In additii?n, the results of the 

Volunteer/Target Involvement Scale^will be presented. The next section of results 

^ill include a verbal description of other analyses of interest. 

Tables 2, 3, and 4 provid^the results of the complex analysis of variance 

for the Adherence to Behavioral Contracting Model scale, the Adherence to Child 

Advocacy. Model scale, and the Adherence to Relationship Skills Model scale, 

respectively. In addition. Figures 1, 2, and 3 pre^^ a graphic description 

of the means for each of the groups by time period. ^^^Re interesting findings 

from these three scales are twofold. In the first instance, it is obvious from 

the analyses of the Adherence to Child Advocacy Model and the Adherence to 

•Behavioral Contracting Model scale, that there were major differences between the 

Child Advocacy/Behavioral Contracting Condition and t^e other three experimental 

groups. In both instances, there are large effects for experimental condition.' 

iB^^lE^f^ii^s^l^^li^and^A^and^^ 
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From Scheffe multiple comparison examination of means, it is . obvious that the , 

maj^it^ of these effects are^^atrribuCed , to differences between the Behavioral 

Contracting/Child Advocacy grpup and tl%, other thi|fe experimental conditions. 



an fact, •-these^ result sy are yccentuat'ed b^ween th^^dvoc^cy/Contrac^p^^ ^■ 
and the* rel>tionshij^ skills gijoup. /This represents a ' ^eionst ration ^ the decree ^ 
to^ which nonprd^^essipnal* volunteers iri^^ondltibn *'a\a1ih^ecl to-^the training Astiper- 

^viEion/yfttervention model fromjihe perspective of program par.ticipants . . Al so 

of interest on '^he Advocacy anc} Contracting scales^are the differentia^' pat terns/ . 

of model adherehce demonstrated by success vs. falAure cases . . In both instances 

XM apjpears^hat failure cases received highe^'evels of the mo^eL intervention 

\ . . \ ' , ^ . - ■ . 

particularly at time periods two and three. In oj^apinlng the relationship between 

■ ^ ' ' ' ' ■ 

these results and outcome recidivism results,- it seems safe to suggest that these 
increased leve is of m*odel adherence are in response to official, recid^^vism among 
cases in those conditions. It- does not appear to-be the case that thes% groups 
received significantly higher levels of modjel intervention prior to ofl'iciall^^ 
recidivating- . ^ r 

Somewh^ more surprising are th^ results on the Relationship Skills scale, 
i^^^s instance, there are no dif ferences between groups that are stat ist ically^ 
reliable. In fact, there is a general increase' (dei(onsf rated by Scheffe multiple 
comparisons) in the relationship scale among alN^ groups regardless of condition 
or success vs. failure distinction. ^ 

While these three scales represented "manipulation checks'" and model adherence 

a number of the other scales reflected additipnal salient dimensrons of 'the inter-' 

vention process. At the most general level was the Volunt^eer /Target Involvement 

Scale reflective of the intensity of the interaction between' the nonprofessional 

volunteet and their assigned student. Table 5 presents the complex analysis of 

Insert Table 5 about here 
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variance for the amount, of time 'i^cale 'ancJo figure 4 presents graphically the means 
fox the same*.dimension. " -As can H^^en f rom Figur'6 5 , ip combination -with the 

■J ' ■ ' 4^ "~ ' ' I' ~ ', ^ ' ■ ' ■ ' 

analysis of. variance"*^ there i.s al geirieral "treind to rSduce ,the amouqjt of time 'spent 
in tlt€ lnter^entii)n^over[the durat/oh of ' the eigj^legn^^^^ interval. This- Is , 

I /■■ / . ^; if"". — ""T" — *^ j-i'i — /• 

' ^ ' * / ^ *'*'^n^er£ Figures /a and 5 about h^re 



most characterized by three out 
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oiytm f our^^fg^^ure 'groups and most dramatically 
'in the relationship failure groub.^pA £|[»^ce6s group^ without exception, show 
a relatively .stable level of i«^^'<^eotili^er time, 

...To summarize the ^^jeS^'WM^^^ here before proceeding to a some- 



what, more general should be no*ted' that both the 



highly specific >an.iplirf;^lMp^^'Je'' '^cales and the most -general amount of 'time sp'ent 
scale produce(J some very unexpected results within a classic experimental" paradigm. 
It has been observed, th^^STodel adherence must necessarily be monit ored*\since 
pne^cannot ^assume that .^alhe generally high levels adherence demanded by experi- 
ment a^ paradigms will iJtecessarijLy^'be adhered to in the. face of real life situa- 
tion^.. In addition/ at the most general level of amount of time sp^nt in inter- 
vention,^ consistency ^s also not demonstrated in the face of of ficial recidivism. 

Additiphal findings of interest from similar analyses of the int^|^ention 
scales were observed in several areas. Op other scales unrelated to the intervention 
mo^el, ..some ir^triguing results ^occurred. In the area of parental involvement 
|a| might be expected, 'thi^ t:iiild Advocacy/Behavioral Contracting group showed the . 
higTiest leVfel of Parental Inv^olv^ment. It should be remembered that these items 
are not necessarily refleCrti^e of Parental Involvement for specific change pur- 
poses, but can include such' 'things .^^s the frequency of talking to €he parents 

and involving them in t>e ge|ieral.' intervention plan without attention to change 
\ ' ;.*)-•.' ^ 

relevant specifics. The other ma±Ti 'finding of interest on the Parental Involvement 

i ' * ' ' ' , 'A 

Dimerisiort is that the rel^t ionsh;ip condition, as would also be expected, 
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di^cmstrated the lowest level of involvement of parents in the /intervention.. 

In the\Recreational Activities scale, a somewhat 'different pattern resul'ted'. 

OA-the one h^m*;^ it is*again the case that the suc;^^^ss groiip showed a moderate ' ^ " 

and stable level-H^ecreational activities 'as does the Chilti Advocacy/Behavioral 

Contracting fai;*re .group. However, other failure groups, most notably in the 

Relationship Condition, show a dramatic decrease over time. Again, in these two 

somewhat generki' areas of description of intervention process, we have seen a 

ft 

good deal of effect for training ^condition and an interaction of intervention 
process with real life events. 

In a somewhat .different area, the legal intervention scale, a rather* dramatic 
pattern of results was demonstrated. All failure groups, with the exception of 
the Relationship Skills failure group/ showed a dramatic and statistically signi- ^ 
f leant increase in Legal System Involvement Scale. The Relationship failure 
group did not demonstrate this pattern. Again, this 'is as might be expected since 
legal intervention was primarily a reactive area for ^ diversion program. ' Specific 
, cally,^hen f aced^ with youth officially recidivating, the volunteers expended a 
good deal of their intervention effort in the direction with dealing with the 
juvenile ^^ustice system. 

Three otl^er salient features of the intervention deserve attention. First, 
in the ar^^^a. oT^^l^amily Intervention: Focus on Changing Parents focused on changing 
the parents, there was a significant effect for group. Again, as might be 
expected, it-was consistently -the case that the Behavioral Contracting/Child 
Advocacy group, regardless of success/failure distinction, showed the highest level 
of Family Intervention Focused on the Parents. Second, in the area of School 

Intervention; Focus on Changing the Youth, a son^^what (iiffcrent pattern of 

s 

results emerged. Again, the Behavioral Contralcting/Ch i 1 d Advocacy group showed 
relatively high levels of involvement in this area as do two of the Low Intensity 
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groups. However, the Low Intensity groups tended to get involved in 

this area only in, reaction to failure.. The Relation- * 
Ship Skills group, as would be expected from training, did not involve themselves 

'* ^ Jit ' i. 

significantly in scJfool intervention. ^In thp third and somewhat different con- ' 
ceptual area. School Intervention Focused on Changing bbi School, an additional 
interesting set of results come fo^h. Again, as would be expected, the Behavioral 
Contracting/Child Advocacy group showed the'highest levels of activity in this area. 
The Relationship group, regardless of success vs, failure distinction, failed 
to involve themselves in school change. to any significant extent. BoCh Low 
Intensity groups, in the face of failure, show minimal levels of involvement in 
School Change activities wh^le the Low' Int ensity success groups do not involve 
themselves in this area at all. 

DISCUSSSION 

Prior to discussion of these results, some introductory comments are ^in 
order. First, it is to be remembered that the results reported are preliminary 
in nature. They are a subset of a large longitudinal study currently being 
conducted by. the authors. As such, they represent the development of a useful 
process monitoring methodology, and have highlighted some salient features of 
intervention process indicative of the need for carrying out such process moni- 
toring assessments. While strict adherence to compTg^ statistical procedures 
and traditional multiple comparison verification ^Tresults has been adhered to, 
the tentativeness of the results reported cannot be overemphasized. 

X The results from the three scales designed to monitor the intervention models 
from the pe|^ctive of program participants indicated that in the case of Adherence to 
Behavioral Contracting and Child Advocacy Model, there was considerable difference 
between experimental conditions. It appears obvious that people receiving 
specific training and supervision in behavioral contracting and child advocacy 
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are in fact the only nonprofessional volunteers to execute these skills to any 
degree. In the case of the Relationship^ Skills scale, a somewhat different 
set of results emerges. ' Namely, there^ did not appear to be significant" differences ' 
between (graining cpnditions on\^s dimension. Two possible explanations for 
this received support from our own experience and from the literature. On the 
^ one hand, it is poss<j.ble that the use of relationship based intervention ^ 
modalities with delinquent groups is extremely difficult. However ,* given the 
level of relationship interventions observed, it seems more plausible that all 
groups were able, even without specific training, to carry out interventions with 
ail acceptable level of relationship skills as an unc^ergirding. The fact that 
essentially untrained college students possess high levels of such skills has 
been demonstrated by other authors (Poser, 1966; Rappaport et al., 1974; Goodman, 
1971). 

The obvious importance of data resulting from such' process scales in monitoring 
^ the intervention model cannot be overemphasized given the above . Within the 
traditional experimental paradigm, given the degree of reasonable contjrol and 
manipulation executed in this study, without such process monitoring^checks , it 
rwould have been assumed that all groups did what they wer^ trained or^ not trained 
to do as specified by the experimental design/ If we are to begin the process of 
understanding the salient processes of intervention programs suth as that described 

h^re, the need for such assessments seems crucial, 

/ 

. The above point is furtlior onniiiasized when the results of a general scale 

such" as the Volunteer/Target Involvement are examined. As was described in the 

. results. $tction, groups differentially trained, particularly In reaction to official 

Y recidivism, tend to show dramatic nMy different levels of time spent in the inter- 

vention. Given the scale leveJ s demonstrated, the range involved is from six to 

eight hours a week to two to three hours a week at various time points by various 
i 

groups. This/ffect was obviously the mo.^t dramatic in the case of the Relntionship 

ERIC 
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condition. The people in the Relationship Skills condition had essentially been 
trained in the development of a "Relationship Skills intervention to be carried 
out between themselves and the youth only. This training was to the exclusion of 
(in fact resti'icted them from) direct interventions in other life areas. JiS 
could be -seen from the other general results described, the Relationship group 
tended not to ^involve themselves in the areas of parents, the home, or school. 
As a result, those who were faced with ^.dlure, essentially stopped carrying out 
the intervention model since it did not fit with the real life events they were 
encountering. 

A second major area exemplified by the study presented here, is the feasibility 
of carrying out i^articipant or consumer oriented assessments which are on the one 
hand open-ended and descriptive in their nature yet on the other hand psychometrical 
sophisticated. A major thrust of the community psychology movement has been the 
importance of examining the processes and effects of social programming from the 
perspective of participants. However, to date, few systematic efforts at the 
development of methodologies for participant or consumer oriented domains have been 
presented. The current study included a highly descriptive approach to the develop- 
ment of such measures. It was clearly demonstrated that such procedures can be 
developed in ways which may ultimately alter the conclusions of evaluations con- 
cerning social programs. At this point in our research, it is safe to say that 
they will enrich the network of information available and allow us to proceed 

more intelligently in (uir exptTincMUal outconio comparisons. 

A final point which needs re-emphasis is the importance of examining the inter- 
play of social programs with real life events. The case in point in the study 
presented here i« the effect of official recidivism (I.e., re involvement 'in tfie 
Juvenile court system) on the actual execution of the intervention model^^l The 
common sense analogy that many of us have experienced during formal training 
in particular social innovations is the response **but it doesn't r(>aily work that 
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way". Some of. the data presented here in fact demonstrate that phenomena 
empirically. In several of the intervention domain areas it was clearly and 
.significantly demonstrated that a critical event, such as official recidivism, 
can dramatically alter a very simple intervention program that is carried out 
between a nonprofessional student volunteer and a single delinquent youth. The 
effects of similar phenomena in other social program areas has only recently 
received attention. It is likely that there is a rich network of relevant infor- 
mation of both an experiential and empi.rical'' nature available in monitoring 
such events. Other aspects of our own research are attempting a more fine grained 
analysis of such phenomena by examining specific life events surrounding the 
referred youth in relationship to the project, its processes, and ultimate success. 

In conclusion, the current study has highlighted the need for particpant 
oriented process monitoring within the context of community oriented social 
innovations. A number of results indicated the degree to which the social 
innovation model was actually carried out, its interaction with critical life 
events, and their relationship to ultimate outcomes. In addition, the current 
study has demonst ratted the feasibility of developing scientifically credible 
methods for j assess in;^ such processes. It is our suspicion that there is consid- 
erable ground wMt urncd ahead of us. 
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Footnotes 

■''Paper presented at annual meetings of the American Psychological 

. Association^Sloronto, Canada, August, 1978 • 
2 
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Obviously the outcome results from this project on many of the other 
variables except those described in this substudy may be of interest to the 
reader. Outcome results on official recidivism, school performance, self- 
report delinquency and a variety of other measures are available from the 

authors upon request. 
- 4 

The intervention monitoring scales consist of a set of approximately 200 
items. Due to the length of this document, they are not included but are avail- 
able upon request from the authors. All items were scaled on five point Likert 
type dimensions and included both estimates of frequency and duration of activity 
queried about. 
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Table 1 



Intervention Survey 



' ' ^' Lack of.^plaints/PositlvejrpwO^»^^n^ This scale reflects the 
extent to which the y^th and volunteer get along and the degree to which there 
ar^ %ot problems in carrying out the intervention. Example item: To what extent 
does the youth like what the volunteer is doing in general?', ^^d "Volunteer can't 
find things the youth likes to do." This scale consists of 23 items and has a 
scale alpha of .86. 



2. Volunteer /Target Involvement . This scale includes three items re- 
flective of the amount of time the volunteer spends carrying out the intervention. 
These three items have a scale alpha of .53. 

3. Parental Involvement. This scale is .aimed at assessing the extent to 
which the parents are part of or included in the intervention and the extent to 
which a relationship was established between the volunteer and the parents. 
Example item: 'To what extent does the volunteer involve the parents in the 
planning of the intervention?" This scale contains 12 items and has a scale alpha 

4. Peer Involvement. This scale reflects the degree to which the friends 
of the youth were included in the intervention. Example item- "How often does 
the volunteer involve the youth's friends In recreational activities'" This scale 
consists of six items which have a scale alpha of .81- 

5. Recreational Activities. This scale reflects the extent of recreational 
activities as a part of the intervention. The basic dimension is the frequency of 
activities that the volunteer and youth engage in together. Example item: "How 
often do the youth and volunteer participate in purchased recreational activities 
together? The scale consists of three items and has a scale alpha of .51. 

6. Family; Focus on Changing Youth. This scale and the next were aimed 

at assessing the extent to which the intervention is concerned with changes in the 
family area This scale reflects how much of that effort is directed at changes 
in the youth s behavior per se. Example item: "To what extent is the intervention 
focused on the youth doing more household chores?" This scale consists of five 
items which have a scale alpha of .71. 

7. Family; Fopus on Changing Parents, This scale has ^ similar purpose 

to number six but is aimed at the extent to which the home intervention was focused 
on getting the parents to do things differently. Example item- "To what extent is 
the intervention focused on getting the Parents to improve their household rules'" 
This scale consists of eight items which have a scale alpha of .86. 

8- School: Focus on Changing Yonth. This scale is ain,ed at assessing 

intervention activities which focus on the school behavior of the youth. The 
basic dimension is the frequency of activity in this area. Example item: "To 
what extent is the volunteer trying to get the youth to do more homework?" This 
scale consists of eight items which have an alpha level of .85. 

9. School; Focus on Changing S chool. This scale is focused on the extent 
to which the volunteer engages in activity aimed at bringing ^bout improvement in 
the school area, but the efforts are directed towards the actions of the school 
st9ff rather than the youth per se. Example item; "How often does the volunteer 
talk to teachers?" This scale consists of ^ive items which have an alpha level 
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10. Jpb Seeking , This scale reflects the extent. to which the intervention 
involves attempts to obtain employment for the youth. The basic dimension was 
frequency of actions or frequency of planning for actions. Example item: "How 
oft en does the volunteer take the youth for job interviews?" This scale consists 
of seven items which have an alpha of .86. 

11* Legal System Involvement . This scale reflects the extent to which 
the volunteer became involved in the juvenile justice system as part of the work 
with the youth. Example item: "Has the volunteer assisted the youth in getting 
a lawyer?" This scale consists of 11 items which have a scale alpha of .77. 

12. Adherence to Behavioral Contracting ^Model . This scale was developed 
to reflect volunteer actions representative of the behavioral contracting model 
of intervention. They represent nine sequential items beginning with assessment 
for contracting and ending with instruction of the youth and significant others 

the contracting method. This scale demonstrates a scale alpha of .95. 

13. Adherence to Child Advocacy Model . This scale consists of items reflec- 
tive of volunteer actions representative of the advocacy model of intervention. 
Namely, to what extent was the volunteer intervening on behalf of the youth to 
gain needed resources. Example item: "To what extent has the volunteer specified 
individuals in control of needed resources?" This scale consists of ten items 
which have a scale alpha of .82. 

1^* Adherence to Relationship Skills Model . This scale consists of seven 
items reflective of relationship skills representative of the relationship model 
of intervention. Namely, to what extent was the volunteer intervening with the 
youth per se in line with the sequential relationship based problem solving model. 
Example item: How much effort does the volunteer put into trying to understand 
the youth's feelings, beliefs, values, etc. The seven items of this scale have 
an alpha of .84. ' 
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Table 2 , 
Analysis of Variance 
Adherence to Behavioral Contracting Model Scale 





DE 


Condition (A) 


3 


Success/Failure (B) 


1 


A X B 


3 


Subjects (C) 




Time (D) 


2 


A X D 


6 


B X D 


2 


A X B X D 


6 ^ 


C X D 


92 



MS 


£ 


Prob 


11.70 


11.56 


<0.0005 






.0^1 




-> . uo 




1.02 






.86 


r 8.8^ 


<0.0005 


.35 


3.58 


.003 


1.^12 


1^1.5^1 


< 0.0005 


.50 ^ 


5.12 


<0.0005 


.01 
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Table 3 
Analysis of Variance 
Adherence to Child Advocacy Model Scale 

■A. 

1 





liE 


MS 


E 


Prob 




XONWTION (A) 

> 


3 


7.50 


^.38 


.009 


.13 


Success/Failure (B) 


1 


1.83 


1.07 


.307 




A X B 


3 


.82 . 


.^8 


.699 




Subjects (C) 


^6 


1.71 








Time (D) 


2 


.006 


.02 


.976 




A X D 


6 


.12 


Al 


.832 




B X D 


2 


.13 


.5^ 


.586 




A X B X D 


6 


.19 


.76 


^ .601 




C X D 


92 


.25 
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Table ^ 
Analysis of Variance 
Adherence to Relationship Skills Model Scale 







. M 


£ 


Prob 


Condition (A) 


3 


' .84 


.500 


.684 


Success/Failure (B) 


1 


.02 


. .01^ 


.905 


A X B 


3 


.45 


.267 


.849 


Subjects (C) 


46 


. .17 






Time (D) 


2 


.80 


6.088 


.003 


A.x D 


6 


.32 


.244 


.961 


B X D 


2 


.22 


.165 


.848 


A x.B X D 


.6 


.46 


.347 


.910 


C X D 


92 


.13 
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Table 6 ^ 
^Analysis of Variance 
Volunteer/Target I nvolvement Scale 



Source 


HE 




£ 


Condition (A) 


3 


1.03^1 - 


.900 


Failure/Success (B) 


1 


.913 


' .79^ 


A X B 


3 


, 2.1^19 


1.870 


Subjects 


46 


1.150 




Time (C) 


2 


4.898 


7.838 


A'x C 


6 


.307 


1.269 


B X C 


2 


1.638 


6.765 


A X B X C 


6 


. .910 


3.759 


Subjects x Time 


92 . 


.2^12 
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FIGURE 3 ^ 
ADHERENCE TO RELATIONSHIP SKILLS MODEL 
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FIGURE 4 
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